Communication between autonomous vehicle and external observers

ABSTRACT

At least one embodiment of this disclosure includes a method for an autonomous vehicle (e.g., a fully autonomous or semi-autonomous vehicle) to communicate with external observers. The method includes: receiving a task at the autonomous vehicle; collecting data that characterizes a surrounding environment of the autonomous vehicle from a sensor coupled to the autonomous vehicle; determining an intended course of action for the autonomous vehicle to undertake based on the task and the collected data; and conveying a human understandable output via an output device, the human understandable output expressly or implicitly indicating the intended course of action to an external observer.

RELATED FIELD

The disclosure relates to autonomous vehicles and in particular tocommunications between autonomous vehicles and humans.

BACKGROUND

Operating a vehicle (e.g., a car, a truck, or a bike) is a challengingtask because a driver has to learn to operate the vehicle, navigatewithin the limitations of traffic laws and the physical environment, andcommunicate with other proximate drivers in that same physicalenvironment (e.g., a road, a parking lot, etc.). Regarding the lastrequirement, human drivers signal their intentions through a number ofintentional and subconscious acts. Some of these signals rely ondevices, which are purposely built into the vehicle, such as turnsignals and brake lights. Other signals rely on innate humancharacteristics. These characteristics include intentional actions, suchas waving a hand to signal another driver to proceed through anintersection and subconscious or reactive actions, such as a driverturning his head before merging lanes. When all other forms of signalinghave failed, human drivers are usually able to speak their intentions orask for assistance, such as when a vehicle breaks down or when a driverbecomes lost.

The signals conveyed are perceived by external observers, such aspedestrians, other drivers, or traffic control officers. The externalobservers are able to interpret these signals to gain insight into adriver's intentions. These insights are important to enable safe andefficient flow of vehicular traffic as well as for providing assistancewhen needed. A central feature of ail of these methods is that anexternal observer need not employ any special equipment to understandthe signal conveyed and most signals do not require any training todiscern.

Advances in autonomous vehicles enable computers or other electronicdevices to drive vehicles. At least two types of autonomous vehiclesexist today—fully autonomous vehicles with no human passengers andsemi-autonomous vehicles that are capable of operating in an autonomousmode while carrying human passengers. As the number of autonomousvehicles increases, there will be a need for better communicationmethods between the autonomous vehicles and external observers.Unfortunately, the signaling methods that are currently built intovehicles, such as turn signals and brake lights, only provide theability to communicate a small subset of the information which isrequired. The signal and brake lights provide a limited ability tocommunicate with external observers. No known current autonomous vehicleprovides a comprehensive means for signaling external observers.

DISCLOSURE OVERVIEW

Broadly speaking, the embodiments disclosed herein describe a number ofdesigns and methods that enable autonomous vehicles to conveynavigation-related intentions to external observers. In one aspect, thisdisclosure describes various ways of conveying information to externalobservers using images displayed on the vehicle. These images may besymbols, words, or pictures. In another aspect, the disclosure describesvarious ways of notifying external observers of the vehicle's intentionto convey information. This notification may be accomplished usinglights or sounds.

The disclosure also describes various ways of conveying information toexternal observers using images projected on the ground in proximity tothe vehicle. These projections may include information concerning theintended trajectory of the vehicle as well as the anti-trajectory (i.e.,an area where the vehicle specifically will not travel).

Additionally, the disclosure describes various ways of conveyinginformation to external observers using an anthropomorphic device. Theanthropomorphic device may be a purpose built object or it may beincorporated as part of a sensor pack already installed on the vehicle.The anthropomorphic device may also be in the form of an image. Finally,the disclosure describes various ways of conveying information toexternal observers using a movement state indicator (e.g., a three-stateindicator).

This Disclosure Overview is provided to introduce a selection ofconcepts in a simplified form that are further described below in theDetailed Description. This Summary is not intended to identify keyfeatures or essential features of the claimed subject matter, nor is itintended to be used to limit the scope of the claimed subject matter. Amore extensive presentation of features, details, utilities, andadvantages of the disclosure is provided in the following writtendescription of various embodiments, illustrated in the accompanyingdrawings, and defined in the appended claims. Some embodiments of thisdisclosure have other aspects, elements, features, and steps in additionto or in place of what is described above. These potential additions andreplacements are described throughout the rest of the specification.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a navigational environment of a vehicle capable of indicatingits navigational intentions to an external observer, in accordance withvarious embodiments.

FIG. 2 is an example of an autonomous vehicle in accordance with variousembodiments.

FIG. 3 is an illustration of an autonomous vehicle displaying a standardtraffic symbol, in accordance with various embodiments.

FIG. 4 is an illustration of an autonomous vehicle displaying an imageof a nearby pedestrian in a cross walk, in accordance with variousembodiments.

FIG. 5 is an illustration of an autonomous vehicle displaying an imageof a traffic officer and a symbol indicating acknowledging a commandfrom the traffic officer, in accordance with various embodiments.

FIG. 6A is an illustration of an autonomous vehicle projecting itsupcoming trajectory on a nearby ground surface using an arrow symbol, inaccordance with various embodiments.

FIG. 6B is an illustration of an autonomous vehicle projecting itsupcoming trajectory on a nearby ground surface using a dash line, inaccordance with various embodiments.

FIG. 6C is an illustration of an autonomous vehicle projecting itsupcoming trajectory on a nearby ground surface using a patternindicating areas that the autonomous vehicle will cover, in accordancewith various embodiments.

FIG. 7 is an illustration of an autonomous vehicle projecting itsupcoming trajectory on a nearby ground surface using an arrow symbol andshowing an oval to indicate an obstacle along the upcoming trajectorythat the autonomous vehicle is aware of, in accordance with variousembodiments.

FIG. 8 is an illustration of an autonomous vehicle projecting an imageof a bar indicating a border beyond which the autonomous vehicle wouldnot traverse, in accordance with various embodiments.

FIG. 9 is an illustration of an autonomous vehicle with a movement stateindicator to convey a human readable output corresponding to an intendedcourse of action, in accordance with various embodiments.

FIG. 10 is a flow chart of a method of operating an autonomous vehicleto convey an intended course of action to an external observer, inaccordance with various embodiments.

FIG. 11 is a block schematic diagram that depicts a machine in theexemplary form of a computer system, within which a set of instructionsfor causing the machine to perform any of the herein disclosedmethodologies may be executed.

The figures depict various embodiments of this disclosure for purposesof illustration only. One skilled in the art will readily recognize fromthe following discussion that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles of the invention described herein.

DETAILED DESCRIPTION

FIG. 1 is a navigational environment 100 of a vehicle 102 capable ofindicating its navigational intentions to an external observer, inaccordance with various embodiments. Present day autonomous vehiclesprovide little, if any, notification of their intentions in adiscernible form to an external observer. This makes it challenging forpeople or systems near such autonomous vehicle to plan its behavior orreact to the behaviors of the autonomous vehicle. The disclosedautonomous vehicle, such as the vehicle 102, solves the above challengeby conveying the intentions of the vehicle 102 in a variety of humanunderstandable forms, using light or sound.

The vehicle 102 can be an autonomous vehicle (e.g., a fully autonomousor a semi-autonomous vehicle) that has an assigned task. The vehicle 102includes a first control system 104, which can be implemented by acomputing device, such as a computer, a field programmable gate array(FPGA), an application specific integrated circuit (ASIC), or otherelectronic devices or systems. The task can be a navigation-related job,in that it requires the vehicle to navigate in some manner, which thevehicle 102 attempts to complete. For example, the task may require thatthe vehicle 102 navigate from a starting point to an ending point (e.g.,a destination address). The ending point may be open ended, in that itmay be changed or amended over time. The task may also require thevehicle 102 to pass a number of way-points along the way or to follow apredefined route. The vehicle 102 is equipped with various sensors thatgather data concerning the vehicle's surroundings. Based on this dataand its assigned task, the vehicle plans an intended course of actionwhich it conveys to external observers in a human understandable form.

FIG. 1 illustrates some of the potential forms of communication that thevehicle 102 is capable of utilizing to convey its intentions to theexternal observers. In one example, the communication can take the formof human driver communication 106, which involves the first controlsystem 104 of the vehicle 102 and a human driver 108. The human drivercommunication 106 can be a light (e.g., a beam, a flash, other radiatingshapes), a sound, a display, an animation, or any combination thereof.The human driver communication 106 can be generated from an outputdevice of the vehicle 102 that is not a conventional signal light orbrake light.

Another example, the communication can fake the form of inter-devicecommunication 112, which involves the first control system 104 and asecond control system 116 of another vehicle 114 (e.g., anotherautonomous vehicle). For example, this type of communication can involvewireless digital or analog communication. For another example, theinter-device communication 112 can be steganographic communication withdigital information encoded into a human understandable presentation(e.g., an image or a sound). The inter-device communication 112 can alsobe other optical based, radio frequency based, or acoustic-basedcommunication with encoded information.

In yet another example, the communication can take the form ofpedestrian communication 120, which involves the first control system104 and a pedestrian 122. The pedestrian 122 can be any externalobserver who is not driving a vehicle. In a further example, thecommunication can take the form of traffic controller communication 126,which involves the first control system 104 and a traffic controller128. The traffic controller 128 can be a person with the authority tocontrol traffic (e.g., a road worker, a police officer, etc.), a trafficcontrol device, or an agent terminal device within the navigationalenvironment 100 for a remote person to control the traffic. Thepedestrian communication 120 and the traffic controller communication126 can take the forms similar to the human driver communication 106,including a light, a sound, a display, an animation, or any combinationthereof. In some embodiments, the first control system 104 cansimultaneously present or generate any of the human driver communication106, the inter-device communication 112, the pedestrian communication120, and the traffic controller communication 126.

FIG. 2 is an example of an autonomous vehicle 200 in accordance withvarious embodiments. The autonomous vehicle 200 may be a fullyautonomous vehicle or a semi-autonomous vehicle. The autonomous vehicle200 includes a control system 202 (e.g., including the first controlsystem 104 of FIG. 1). The control system 202 determines (e.g.,calculates and plans) a trajectory for the autonomous vehicle 200 andexecute the trajectory in accordance with the traffic laws and physicallimitations of its surrounding environment, such as the navigationalenvironment 100 of FIG. 1. The autonomous vehicle 200 further includesan intention communication subsystem 204. In some embodiments, theintention communication subsystem 204 is physically integrated with thecontrol system 202. In other embodiments, the intention communicationsubsystem 204 is a separate device coupled to the control system 202,via a wire or wirelessly. In some embodiments, the control system 202and/or the intention communication subsystem 204 can be detachable fromthe autonomous vehicle 200 and/or can be portable.

The intention communication subsystem 204 is able to generate a humanunderstandable output to convey an intended course of action (e.g., thetrajectory) of the autonomous vehicle 200. The intended course of actioncan be explicitly illustrated or implicitly illustrated (e.g., byacknowledging a traffic law, an obstacle, or a physical border beyondwhich the autonomous vehicle 200 will not traverse). For example, thecontrol system 202 can determine the intended course of action based onone or more user configurations (e.g., by setting a destination) througha configuration interface 206. The control system 202 can also take intoaccount the surrounding environment of the autonomous vehicle 200 basedon sensor inputs from one or more sensors 210. Based on the sensorinputs and the user configurations, the control system 202 is then ableto determine the trajectory and route to be taken by the autonomousvehicle 200 and any additional actions needed to navigate to thedestination (e.g., avoiding obstacles, avoiding violations of trafficlaw, reach intermediary waypoints, etc.).

The sensors 210 can include one or more cameras 212, one or moremicrophones 214, one or more radars 216, one or more sonars 218, one ormore lidars 220, one or more radio frequency (RF) antennas 222 (as asensor and/or a transceiver for machine readable communication), one ormore tactile sensors 224, other types of passive or active sensors, orany combination thereof. The tactile sensors 224 can detect contactbetween an external object (e.g., a person or an object) and theautonomous vehicle 200. For example, the tactile sensors 224 can beforce sensors, accelerometers, or touchscreen sensors. The sensors 210collect data describing the surrounding environment of the autonomousvehicle 200 and/or providing self-diagnosis feedback of the autonomousvehicle 200 (e.g., relative to the surrounding environment). The controlsystem 202 can also record metadata from the sensors 210 for furtheranalysis. For example, such analysis includes the control system 202detecting various events, conditions, and objects through the collecteddata, including detecting traffic events, accidents, pedestrians, fellowdrivers, fellow vehicles, active or passive traffic signs, roadconditions, weather conditions, or any combination thereof. The controlsystem 202 can further detect characteristics of such events or objects,including types of fellow vehicles, relative locations of the detectedobjects, anticipated behaviors of the detected objects, locations ofsensors of detected devices, locations of perceptive organs (e.g., eyesor ears) of detected people, or any combination thereof.

The intention communication subsystem 204 expresses and conveys thehuman understandable outputs through one or more output devices 230. Theoutput devices 230 includes one or more projectors 232, one or moredisplay devices 234, one or more speakers 236, one or more lights 238,one or more mechanically actuated devices 240, or any combinationthereof. The output devices 230 can be unidirectional, bidirectional,multidirectional, or omnidirectional.

When conveying a human understandable output using the output devices230, the intention communication subsystem 204 can direct the one ormore of the output devices 230 at a detected external observer or anarea that the detected external observer can perceive. In variousembodiments, the intention communication subsystem 204 can present thedetected objects (e.g., detected based on the sensor data) in the humanunderstandable output to acknowledge the control system 202's awarenessof the obstacle.

A user of the autonomous vehicle 200 can further configure the humanunderstandable output, such as the form of the output, through theconfiguration interface 206. For example, the user can choose the bestform of communication to express and convey an intended course of actionas determined by the control system 202. The configuration interface 206can present two or more options for conveying the intended course ofaction through the output devices 230. For example, below are severalexamples of human understandable outputs that can be used to expresslyor implicitly indicate the intended course of action of the autonomousvehicle 200.

Display on Vehicle

The human understandable output can be conveyed using a number ofdifferent display formats via the display devices 234. The displaydevices 234 can be a surface display or other displays mounted on theautonomous vehicle 200. The display devices 234 can be retrofitted anddetachably coupled to the autonomous vehicle 200. In one format, theintentions of the control system 202 can be conveyed using images. Theintention communication subsystem 204 can select these images from alist of standardized transportation or traffic symbols. These symbolscan be displayed on the autonomous vehicle 200's windshield or othersurface. When no such standardized symbol exists for conveying thenecessary information, words can be displayed instead. For example, FIG.3 is an illustration of an autonomous vehicle 300, such as theautonomous vehicle 200, displaying a standard traffic symbol 302 on adisplay 304 visible through a windshield of the autonomous vehicle 300,in accordance with various embodiments.

In another format, the image displayed can also be a photo of a nearbyobstacle or observer. For example, if the autonomous vehicle 200approaches a cross walk, the intention communication subsystem 204 candisplay a photo of a pedestrian in the cross walk next to a green light.This conveys that the autonomous vehicle 200 has recognized the presenceof the pedestrian, and that the autonomous vehicle 200 intends to yieldto the pedestrian. For example, FIG. 4 is an illustration of anautonomous vehicle 400, such as the autonomous vehicle 200, displayingan image 402 of a nearby pedestrian in a cross walk, in accordance withvarious embodiments. The autonomous vehicle 400 includes a display 404that presents the image 402 of the pedestrian, an image 406 of the crosswalk, and a symbol 408 representing that the autonomous vehicle 200intends to yield.

In another example, the intention communication subsystem 204 candisplay an image of an external observer next to a check mark, such as agreen check mark. This would signal to the external observer that theautonomous vehicle 200 has recognized a command issued by the externalobserver. The command can be visual, auditory, or radio frequency. Thiscommand can originate from a traffic officer instructing the autonomousvehicle 200 to proceed through an intersection or to pull to the side ofthe road. For example, FIG. 5 is an illustration of an autonomousvehicle 500 (e.g., the autonomous vehicle 200) displaying an image 502of a traffic officer and a symbol 504 on a display 506 indicatingacknowledging a command from the traffic officer, in accordance withvarious embodiments.

Display on Ground or Other Surface

Rather than display the above human understandable output on a surfacedisplay of the autonomous vehicle 200, the intention communicationsubsystem 204 can project, via the one or more projectors 232, theimages or words on the ground or other surfaces, such as buildings orother vehicle exteriors, in proximity to the autonomous vehicle 200.This projection can be accomplished using a laser projector or similarprojector capable of producing an image viewable under daylightconditions. This projection may also use a light manipulation device,instead of a light generating device, such as an apparatus comprisingone or more mirrors, one or more lenses, one or more filters, one ormore diffusers, one or more diffraction grids, or any combinationthereof. Such light manipulation device can harness the ambient daylight to convey the human understand able output.

In addition to displaying the above information, the humanunderstandable output can also indicate one or more aspects of theintended course of action of the autonomous vehicle 200, such as theintended trajectory. This trajectory can be projected in an abstractedor simplified form. For example, an arrow projected on the ground infront of the autonomous vehicle 200 conveys the intended course ofaction as determined by the control system 202. The direction and shapeof this arrow can be altered as needed, so that an arrow that curves tothe right would indicate that the autonomous vehicle 200 intends to turnright. If the autonomous vehicle 200 intends to turn right, but notimmediately, the bar portion (rectangular portion of an arrow on whichthe triangular portion is affixed) can be elongated. In someembodiments, the intention communication subsystem 204 can project acountdown (e.g., numeric or symbolic) of when to execute the trajectory.For example, FIG. 6A is an illustration of an autonomous vehicle 600(e.g., the autonomous vehicle 200) projecting its upcoming trajectory ona nearby ground surface using an arrow symbol 602, in accordance withvarious embodiments.

Alternatively, the trajectory projected can be projected in an absoluteform. In this form, the trajectory projected depicts the exact routethat the autonomous vehicle 200 intends to follow. For example, whenparking the trajectory can be displayed as a dashed line, which curvesinto a parking spot. For another example, FIG. 6B is an illustration ofan autonomous vehicle 650 (e.g., the autonomous vehicle 200) projectingits upcoming trajectory on a nearby ground surface using a dash line652, in accordance with various embodiments. For yet another example,FIG. 6C is an illustration of an autonomous vehicle 670 (e.g., theautonomous vehicle 200) projecting its upcoming trajectory on a nearbyground surface using a pattern 672 indicating areas that the autonomousvehicle 670 will cover, in accordance with various embodiments.

Additionally, the intended trajectory projected on the ground caninclude an obstacle symbol, such as a notch indicating the location of adetected obstacle. For example, if the autonomous vehicle 200 intends toturn right at an intersection, but recognizes that a pedestrian is inthe crosswalk, the autonomous vehicle 200 can project a right arrow onthe ground with a notch cut out of the triangular portion of the arrow,to indicate that the autonomous vehicle 200 recognizes the existence ofan obstacle in its trajectory. This notch can be highlighted by usinganother color, such as red. As the pedestrian proceeds to cross thestreet, the notch correspondingly moves across the arrow until thepedestrian is no longer in the trajectory. At this point, the autonomousvehicle 200 can continue to project a circle on the ground near thearrow to indicate that the autonomous vehicle 200 recognizes a nearbyobstacle but that the autonomous vehicle 200 believes the obstacle to besafely outside the intended trajectory. For example, FIG. 7 is anillustration of an autonomous vehicle 700 (e.g., the autonomous vehicle200) projecting its upcoming trajectory on a nearby ground surface usingan arrow symbol 702 and showing an oval 704 to indicate an obstaclealong the upcoming trajectory that the autonomous vehicle 700 is awareof, in accordance with various embodiments. The autonomous vehicle 700can project with a projector 706 mounted on its top cover near thewindshield.

Similar to displaying a notch, the intended trajectory projected on theground can include an “anti-trajectory” or exclusion zones that theautonomous vehicle 200 intends to avoid. In one example, theanti-trajectory is based on the predicted motion of detected obstacles,such as if the autonomous vehicle 200 detects a pedestrian, via thesensors 210, standing on the corner waiting to cross the street, a largered line can be projected across the path of the autonomous vehicle 200to indicate that the autonomous vehicle 200 will not proceed forwarduntil the pedestrian crosses the street. For example, FIG. 8 is anillustration of an autonomous vehicle 800 (e.g., the autonomous vehicle200) projecting an image 802 of a bar indicating a border beyond whichthe autonomous vehicle 800 would not traverse, in accordance withvarious embodiments. Alternatively the autonomous vehicle 200 whenturning left can project a green left arrow with a red line above thearrow indicating that the vehicle will not travel straight through theintersection.

The trajectory projected on the ground can only display a limited amountof information due to limitations inherent in projecting on the ground.To make the most of the available projection area the informationprojected can be limited based on time or distance. For example, onlythe trajectory information for the next three seconds is displayed onthe ground. The parameter of this time limitation may be configurablethrough the configuration interface 206. Alternatively, the trajectoryinformation for the next 100 feet is displayed on the ground. Theparameter of this distance limitation may be configurable through theconfiguration interface 206.

The intended trajectory can also be projected as a pattern. The patternis comprised of both solid and void portions, such as a Crosshatchpattern. The void portions have no light projected on them. This allowsfor overlapping trajectories to be detected. For example, if theautonomous vehicle 200 is present at each of the entrances to a four wayintersection, the center of the resulting overlapping trajectoriesprojected on the ground will be nearly solid at the center of theintersection. As a result, the “solidness” of the pattern projected onthe ground can be used to determine the number of autonomous vehicleswhich are expected to traverse a particular location. Similarly, theincreased intensity of the light would also denote that multiplevehicles are expected to traverse a particular location. In variousembodiments, the autonomous vehicles can communicate with each otherusing non-human understandable communication, such as radio frequencycommunication via the RF antennas 222.

Movement State Indicator

The autonomous vehicle 200 can also convey its intentions using amovement state indicator, such as a three color indicator. Any threecolors can be used, but the common scheme of green, yellow, and redlights may be the most easily understood. The lights can indicate anynumber of different things. In one embodiment, the green light indicatesthat the autonomous vehicle 200 is operating normally and does notdetect any dangers. The yellow light indicates that the autonomousvehicle 200 is still operating but detects nearby obstacles which may becausing the autonomous vehicle 200 to operate more slowly orerratically. Finally, the red light indicates that an obstacle or eventis preventing the autonomous vehicle 200 from proceeding at all. Thisthree color indicator may be an object mounted to the autonomous vehicle200, or alternatively the indicator may be incorporated in the imageprojected on the ground in proximity to the autonomous vehicle 200. Forexample, FIG. 9 is an illustration of an autonomous vehicle 900 (e.g.,the autonomous vehicle 200) with a state indicator 902 to convey a humanreadable output corresponding to an intended course of action, inaccordance with various embodiments.

Anthropomorphic Device

In another embodiment, the autonomous vehicle 200 conveys its intentionsusing an anthropomorphic device. This anthropomorphic device conveysintentions using actions, motions, or gestures that are similar to thosethat a human would perform.

This anthropomorphic device may be a sensor pack mounted in the locationnormally occupied by one of the rear view mirrors in a conventionalvehicle. The anthropomorphic device has a pan and tilt head that is usedto convey the intentions of the autonomous vehicle 200 to externalobservers by panning and tilting in the direction that the autonomousvehicle 200 intends to travel. For example, when stopped at anintersection, the anthropomorphic device can pan to “look” right toconvey an intention to turn right. Similarly, the anthropomorphic devicecan pan from left to right when stopped at an intersection to convey animpression that the autonomous vehicle 200 is checking for crossingtraffic. At freeway speeds, the anthropomorphic device can pan to theright to convey the control system 202's intention to merge to theright.

In some embodiments, the anthropomorphic device implements a cartoonhand. For example, the cartoon hand can be a mechanically actuateddevice. For another example, one of the output devices 230 may be usedto project or display the cartoon hand on a surface on the autonomousvehicle 200. Alternatively, one of the output devices 230 may be used toproject the cartoon hand on the ground in proximity to the autonomousvehicle 200. The cartoon hand can be used to convey instructions toexternal observers. For example, at a four way stop, the autonomousvehicle 200 can signal another vehicle to proceed through theintersection by projecting a waving hand on the windshield of theautonomous vehicle 200, in a manner similar to a human waving anotherdriver through the intersection. The cartoon hand can also point tospecific obstacles to acknowledge the existence of the obstacle. Theintention communication subsystem 204 can also signify the receipt of acommand from an external observer by displaying the thumbs up sign.

In other embodiments, the anthropomorphic device implements an animatedface with eyes. The intention communication subsystem 204 can implementthe face using one of the output devices 230. The face can be projectedor displayed on a surface of the autonomous vehicle 200 or on the groundin proximity to the autonomous vehicle 200. The face can convey theintended travel direction (according to the control system 202) of theautonomous vehicle 200 by moving the eyes in the direction of theanticipated travel.

External Observer Confirmation

In some instances, the autonomous vehicle 200 will need to request andconfirm the attention of an external observer. This can be necessarybecause the autonomous vehicle 200 requires assistance from the externalobserver or the autonomous vehicle 200 wants to ensure that the externalobserver is aware of the existence of the autonomous vehicle 200. Toachieve this, the autonomous vehicle 200 first performs a notificationstep to attract the attention of an external observer. This notificationstep is performed using an attention getting device (e.g., one of theoutput devices 230).

In one embodiment, the attention getting device is a notificationdisplayed on the ground in proximity to the autonomous vehicle 200. Forexample, the autonomous vehicle 200 can project a bright red flashinglight on the ground around the autonomous vehicle 200. In anotherembodiment, the autonomous vehicle 200 can use an outward projectinglight to gain the attention of external observers by projecting a lighttowards the face of an external observer. The intention communicationsubsystem 204 can analyze the sensor data to confirm that the particularexternal observer is paying attention to the autonomous vehicle 200.

After gaining the external observer's attention, the autonomous vehicle200 then conveys and presents the human understandable output, and canrequest that an action be taken by an external observer in responsethereto. For example, the intention communication subsystem 204 can askan external observer to stand on the sidewalk by displaying a writtencommand on the exterior of the windscreen such as “Please step onto thesidewalk.”

The notification may not always be directed at a particular observer orrequire a response from an external observer. For example, the outputdevices 230 can include a light bar on its roof similar to those used bytaxi cabs that can be used to indicate a specific aspect of theautonomous vehicle 200. In one example, this light bar indicates thatthe autonomous vehicle 200 is in a convoy. This indication is necessaryfor the safety of non-autonomous vehicles that may be traveling on thesame roadway as the convoy. Without the notification, non-autonomousvehicles may attempt to merge into the convoy. In another example, thelight bar can be used to notify nearby semi-autonomous vehicles that theautonomous vehicle 200 desires to enter into a convoy, thus enablingother drivers to navigate into position to form a convoy.

Embedded Data

In each of the above examples, the intention communication subsystem 204presents a human understandable output. In some embodiments, the humanunderstandable output also encodes a machine readable component. Forexample, the state indicator described above can strobe its light at ahigh rate, above 16 hertz, that is imperceptible to humans to encodemachine readable information. A specialized sensor is capable ofdiscerning the embedded information. This specialized sensor can becontained within a handheld device, such as a tablet or smartphone, thatcan be used to display the additional information to the user. A trafficofficer or other roadway manager can use such a device to influence,direct, or communicate with the autonomous vehicle 200. In someembodiments, the specialized sensor is embedded in other autonomousvehicles, allowing that autonomous vehicle 200 to receive theinformation. This embedded information can contain additional detailsconcerning the autonomous vehicle 200's trajectory, state, orintentions.

At least some of the components and/or modules associated with theautonomous vehicle 200 can be implemented in the form of special-purposecircuitry, or in the form of one or more appropriately programmedprogrammable processors, or a combination thereof. For example, themodules described can be implemented as instructions on a tangiblestorage memory capable of being executed by a processor or a controlcircuitry. The tangible storage memory may be volatile or non-volatilememory. In some embodiments, the volatile memory may be considered“non-transitory” in the sense that it is not transitory signal. Modulesmay be operable when executed by a processor or other computing device,e.g., a single board chip, application specific integrated circuit, afield programmable gate array, a wireless network capable computingdevice, or any combination thereof.

Each of the modules and/or components may operate individually andindependently of other modules or components. Some or all of the modulesmay be executed on the same host device or on separate devices. Theseparate devices can be coupled together through one or morecommunication channels (e.g., wireless or wired channel) to coordinatetheir operations. Some or all of the components and/or modules may becombined as one component or module.

A single component or module may be divided into sub-modules orsub-components, each sub-module or sub-component performing separatemethod step or method steps of the single module or component. In someembodiments, at least some of the modules and/or components share accessto a memory space. For example, one module or component may access dataaccessed by or transformed by another module or component. The modulesor components may be considered “coupled” to one another if they share aphysical connection or a virtual connection, directly or indirectly,allowing data accessed or modified from one module or component to beaccessed in another module or component. In some embodiments, at leastsome of the modules can be upgraded or modified remotely. The autonomousvehicle 200 may include additional, fewer, or different modules forvarious applications.

FIG. 10 is a flow chart of a method 1000 of operating an autonomousvehicle (e.g., the autonomous vehicle 200 of FIG. 2) to convey anintended course of action to an external observer, in accordance withvarious embodiments. The autonomous vehicle can be driven by a controlsystem (e.g., the control system 202 of FIG. 2). The method 1000includes the control system receiving a task (e.g., a navigation task)for the autonomous vehicle at step 1002. For example, the task can be todrive to a particular destination, to undertake a goal that requiresmovement (e.g., to pick up trash, to obtain fuel, to join a convoy), tosecure the autonomous vehicle (e.g., to lock the door or start an alarmunder certain circumstances, etc.), to influence the autonomousvehicle's environment (e.g., to request removal of an obstacle in thesurround environment or to request assistance in refueling), or othervehicle-related, location-based, or navigation-related tasks.

At step 1004, the control system collects data that characterizes asurrounding environment of the autonomous vehicle using a sensor (ormultiple sensors, such as the sensors 210 of FIG. 2) coupled to thecontrol system. Data is collected in real-time while the autonomousvehicle is operative (e.g., both when moving and when stationary). Atstep 1006, the control system determines an intended course of action todrive the autonomous vehicle utilizing the control system based on thetask assigned to the control system and the collected data related tothe surrounding environment.

At step 1008, the control system or an intention communication system(e.g., the intention communication subsystem 204 of FIG. 2) detects anobject in the surrounding environment and a characteristic of the objectbased on the collected data from the sensor. For example, step 1008 caninclude detecting presence of an external observer or an obstacle basedon the collected data. Step 1008 can include detecting a location of theexternal observer or the obstacle relative to the location of theautonomous vehicle.

At step 1010, the control system produces an attention grabbing effectvia a first output device to notify the external observer (e.g., eithera specific external observer detected in step 1008 or any externalobserver in proximity to the autonomous vehicle) of an intent to conveya message. For example, producing the attention grabbing effect caninclude displaying a notification on a ground surface in proximity tothe autonomous vehicle. The attention grabbing effect can be produced bya light manipulation apparatus or a light producing apparatus. In someembodiments, step 1008 includes detecting a face of the externalobserver based on the collected data. Step 1010 can then includeproducing the attention grabbing effect by projecting light towards adirection corresponding to the detected face of the external observer.

At step 1012, the intention communication system conveys a humanunderstandable output (e.g., as the intended message) via a secondoutput device (e.g., one of the output devices 230 of FIG. 2). In someembodiments, the second output device has more than two states, unlike aconventional turn signal or brake light. The first output device can bethe same or different from the second output device. The humanunderstandable output indicates the intended course of action of theautonomous vehicle to the external observer.

Conveying the human understandable output can include targeting thehuman understandable output at a detected location of the externalobserver utilizing the second output device that is directional.Conveying the human understandable output can include conveying any of astandardized transportation symbol, a human language word, or an imageon a display of the autonomous vehicle. The image can be an image of theexternal observer. For example, step 1008 can optionally include thecontrol system detecting a gesture-based command from the externalobserver to the autonomous vehicle based on the collected data (e.g., byrunning a gesture detecting process on video or photographs of thesurrounding environment). The image of the external observer (as theissuer of the gesture-based command) can be used to convey anacknowledgement of the receipt and approval of the gesture-basedcommand. Conveying the human understandable output can also includeconveying using a sound producing apparatus with a sound clip as thehuman understandable output or projecting the human understandableoutput via a projector to a ground surface in proximity to theautonomous vehicle.

In some embodiments, step 1006 can include determining to make a requestthe external observer. In that case, step 1012 can include presentingthe request to the external observer. The autonomous vehicle can presentthe request generally via at least one of its output devices or targetat least one of the output devices at the specific external observer.

In some embodiments, conveying the human understandable output includesindicating that the autonomous vehicle is in a convoy. For example, step1006 can include determining that the autonomous vehicle is to enterinto a convoy; and step 1012 can include indicating, via the humanunderstandable output, that the autonomous vehicle is waiting for orready to enter into the convoy.

Step 1008 can optionally include the control system detecting apotential obstacle in proximity to the autonomous vehicle. The potentialobstacle can be the external observer, who is located in substantialproximity to a trajectory of the autonomous vehicle, based on thecollected data that characterizes the surrounding environment.

In various embodiments, step 1012 includes indicating one or moreaspects of the intended course of action via the human understandableoutput. For example, the aspects can include an intended trajectory ofthe autonomous vehicle or a location of a potential or detectedobstacle. The intended trajectory can be presented as an arrow, a dashline, or some other pattern. For example, the intention communicationsystem can determine an extent (e.g., length of the arrow or the dashedline) of the intended trajectory based on a predetermine time interval.The intended trajectory can be indicated using a laser or a projector.The pattern of the intended trajectory can include voids such that, whenmultiple vehicles project onto a same location, a resulting pattern ofoverlapping projections has fewer or smaller voids.

For another example, the aspects can include an avoidance area that theautonomous vehicle intends to avoid. The avoidance area can be determinein step 1006. For example, determining the intended course of action caninclude detecting an obstacle based on the collected data anddetermining the avoidance area by estimating a predicated motion of thedetected obstacle.

In some embodiments, the second output device is an anthropomorphicdevice configured to convey the human understandable output. Theanthropomorphic device can be a mechanically actuated device (e.g., themechanically actuated devices 240 of FIG. 2). For example, theanthropomorphic device can be a sensor pack mounted in a locationnormally occupied by a rear view mirror in a conventional automobile.The anthropomorphic device can be a camera capable of pan, tilt, andzoom. The anthropomorphic device can be a cartoon hand.

The anthropomorphic device can be a display (e.g., one of the displaydevices 234) or a projector (e.g., one of the projectors 232) emulatinga person or part of a person (e.g., by animating a face). The animatedface can include animated eyes. The anthropomorphic device can alsopresent an animation of any other body parts, including a hand or ahead. For example, the anthropomorphic device can be configured toperform a gesture as the human understandable output. The gesture, forexample, can indicate that the autonomous vehicle recognizes an obstacleor that the autonomous vehicle acknowledges a command (e.g., thegesture-based command) received from the external observer. The gesturecan also indicate an intended direction of travel as determined by thecontrol system.

In some embodiments, the human understandable output is a stateindicator (e.g., a three color state indicator). The state indicate canindicate movement or navigation state of the control system. Forexample, the human understandable output is selected from one of threepossible states, such as red for stopping, green for moving normally,and yellow for moving slowly. The state indicator can be produced by alight producing device or a projector. The state indicator can bepresented on the autonomous vehicle or on a ground surface in proximityto the autonomous vehicle (e.g., by projection). The state indicator cantake the form of a line indicating that the autonomous vehicle will notcross the line.

In some embodiments, conveying the human understandable output alsoincludes embedding machine readable information in the humanunderstandable output. The machine readable information can conveyadditional information beyond which is apparent to a human observer. Theintention communication system can embed the machine readableinformation by pulse-coding the human understandable output. Thismachine readable information is useful to provide additional informationto a mobile device of an external observer. This machine readableinformation is also useful in helping observers with disabilities. Forexample, blind pedestrians can use an electronic cane. The electroniccane can convert the machine readable information to audio informationor tactile information.

In some embodiments, step 1008 can include detecting external observerswith disabilities, such as by detecting presence of a white cane forvisually impaired persons. In step 1012, the intention communicationsystem can select the form of the human understandable output to betterserve an external observer with a detected disability.

While processes or blocks are presented in a given order, alternativeembodiments may perform routines having steps, or employ systems havingblocks, in a different order, and some processes or blocks may bedeleted, moved, added, subdivided, combined, and/or modified to providealternative or subcombinations. Each of these processes or blocks may beimplemented in a variety of different ways. Also, while processes orblocks are at times shown as being performed in series, these processesor blocks may instead be performed in parallel, or may be performed atdifferent times.

FIG. 11 is a block schematic diagram that depicts a machine in theexemplary form of a computer system 1100, within which a set ofinstructions for causing the machine to perform any of the hereindisclosed methodologies (e.g., FIG. 10) may be executed. For example,the control system 202 or the intention communication subsystem 204 ofFIG. 2 can be the computer system 1100. In some embodiments, thecomputer system 1100 may include a network router, a network switch, anetwork bridge, personal digital assistant (PDA), a cellular telephone,a Web appliance or any machine capable of executing or transmitting asequence of instructions that specify actions to be taken. The computersystem 1100 is intended to illustrate a hardware device on which any ofthe instructions, processes, modules and components depicted in thefigures above (and any other processes, techniques, modules and/orcomponents described in this specification) can be implemented. Asshown, the computer system 1100 includes a processor 1102, memory 1104,non-volatile memory 1106, and a network interface 1108. Various commoncomponents (e.g., cache memory) are omitted for illustrative simplicity.The computer system 1100 can be of any applicable known or convenienttype, e.g., a personal computer (PC), server-class computer or mobiledevice (e.g., smartphone, card reader, tablet computer, etc.). Thecomponents of the computer system 1100 can be coupled together via a busand/or through any other known or convenient form(s) of interconnects).

One of ordinary skill in the relevant art will recognize that the terms“machine-readable (storage) medium” or “computer-readable (storage)medium” include any type of device that is accessible by the processor1102. The memory 1104 is coupled to the processor 1102 by, for example,a bus 1110. The memory 1104 can include, by way of example but notlimitation, random access memory (RAM), e.g., dynamic RAM (DRAM) andstatic RAM (SRAM). The memory 1104 can be local, remote, or distributed.

The bus 1110 also couples the processor 1102 to the non-volatile memory1106 and drive unit 1112. The non-volatile memory 1106 may be a harddisk, a magnetic-optical disk, an optical disk, a read-only memory(ROM), e.g., a CD-ROM, Erasable Programmable Read-Only Memory (EPROM),or Electrically Erasable Programmable Read-Only Memory (EEPROM), amagnetic or optical card, or another form of storage for large amountsof data. The non-volatile memory 1106 can be local, remote, ordistributed.

The data structures, modules, and instruction steps described in thefigures above may be stored in the non-volatile memory 1106, the driveunit 1112, or the memory 1104. The processor 1102 may execute one ormore of the modules stored in the memory components.

The bus 1110 also couples the processor 1102 to the network interface1108. The network interface 1108 can include one or more of a modem ornetwork interface. A modem or network interface can be considered to bepart of the computer system 1100. The network interface 1108 can includean Ethernet card, a Bluetooth card, an optical fiber interface, a cablemodem, a token ring interface, or other interfaces for coupling acomputer system to other computer systems.

It is to be understood that embodiments may be used as or to supportsoftware programs or software modules executed upon some form ofprocessing core (e.g., the CPU of a computer) or otherwise implementedor realized upon or within a machine or computer readable medium. Amachine-readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine, e.g., acomputer. For example, a machine readable medium includes read-onlymemory (ROM); random access memory (RAM); magnetic disk storage media;optical storage media; flash memory devices; electrical, optical,acoustical or other form of propagated signals, for example, carrierwaves, infrared signals, digital signals, etc.; or any other type ofmedia suitable for storing or transmitting information.

Some embodiments of the disclosure have other aspects, elements,features, and steps in addition to or in place of what is describedabove. These potential additions and replacements are describedthroughout the rest of the specification.

What is claimed is:
 1. A method for an autonomous vehicle to communicatewith external observers, comprising: receiving a task at the autonomousvehicle; collecting data that characterizes a surrounding environment ofthe autonomous vehicle from a sensor coupled to the autonomous vehicle;determining an intended course of action for the autonomous vehicle toundertake based on the task and the collected data; projecting a humanunderstandable output, via a projector that manipulates or produceslight, to a ground surface in proximity to the autonomous vehicle; andwherein the human understandable output indicates the intended course ofaction of the autonomous vehicle to an external observer; and whereinthe autonomous vehicle confirms recognition of the external observer bypresenting, via the projector, an image of the external observer.
 2. Themethod of claim 1, wherein projecting the human understandable outputincludes indicating an intended trajectory of the autonomous vehicle. 3.The method of claim 2, further comprising detecting a location of apotential obstacle from the collected data; and wherein the intendedtrajectory includes a notch that indicates the detected location of thepotential obstacle.
 4. The method of claim 2, wherein indicating theintended trajectory includes displaying an extent of the intendedtrajectory based on a predetermined time interval.
 5. The method ofclaim 2, wherein the intended trajectory is indicated using a laser. 6.The method of claim 2, wherein indicating the intended trajectoryincludes projecting the intended trajectory as a pattern.
 7. The methodof claim 6, wherein the pattern includes voids such that, when multiplevehicles project onto a same location, a resulting pattern ofoverlapping projections has fewer or smaller voids.
 8. The method ofclaim 1, wherein projecting the human understandable output includesindicating an avoidance area that the autonomous vehicle intends toavoid.
 9. The method of claim 8, wherein determining the intended courseof action includes: detecting an obstacle based on the collected data;and determining the avoidance area by estimating a predicted motion ofthe detected obstacle.
 10. A vehicle comprising: a control system withelectronic circuitry configured to enable the vehicle to complete a taskautonomously or semi-autonomously; a sensor, coupled to the controlsystem, configured to collect data related to a surrounding environmentof the vehicle; wherein the control system is configured to determine anintended course of action for the vehicle based on the task and thecollected data; and an anthropomorphic device, coupled to the controlsystem, configured to convey a human understandable output regarding theintended course of action to an external observer, wherein the vehicleconfirms recognition of the external observer by presenting, via theanthropomorphic device, an image of the external observer.
 11. Thevehicle of claim 10, wherein the anthropomorphic device is configured tomove in a pattern or a human understandable gesture to convey the humanunderstandable output.
 12. The vehicle of claim 11, wherein theanthropomorphic device is a sensor pack mounted over an area between adriver seat and a passenger seat.
 13. The vehicle of claim 12, whereinthe anthropomorphic device is a camera capable of pan, tilt, and zoom.14. The vehicle of claim 10, wherein the anthropomorphic device is anactuated hand or a display device configured to animate a cartoon hand.15. The vehicle of claim 10, wherein the anthropomorphic device is adisplay device configured to animate a face with eyes.
 16. The vehicleof claim 10, wherein the anthropomorphic device is configured toindicate recognition of an obstacle.
 17. The vehicle of claim 10,wherein the anthropomorphic device is configured to indicateacknowledgement of a command received from the external observer. 18.The vehicle of claim 10, wherein the anthropomorphic device isconfigured to indicate an intended direction of travel of the vehicle.19. A method for an autonomous vehicle to communicate with externalobservers, comprising: receiving a task at the autonomous vehicle;collecting data that characterizes a surrounding environment of theautonomous vehicle from a sensor coupled to the autonomous vehicle;determining an intended course of action for the autonomous vehicle toundertake based on the task and the collected data; and conveying ahuman understandable output, via an output device, wherein the humanunderstandable output indicates the intended course of action of theautonomous vehicle to an external observer, wherein the autonomousvehicle confirms recognition of the external observer by presenting, viathe output device, an image of the external observer.
 20. The method ofclaim 19, further comprising detecting a location of the externalobserver based on the collected data; and wherein conveying the humanunderstandable output includes targeting the human understandable outputat the detected location of the external observer.
 21. The method ofclaim 19, wherein the output device conveying the human understandableoutput is a sound producing apparatus.
 22. The method of claim 19,wherein the output device conveying the human understandable outputdisplays at least one of a standardized transportation symbol, a humanlanguage word, and an image on a display of the autonomous vehicle. 23.The method of claim 19, wherein the collected data that characterizesthe surrounding environment includes detecting a command from theexternal observer to the autonomous vehicle, wherein the image of theexternal observer indicates acknowledgement of the command.
 24. Themethod of claim 19, further comprising producing an attention grabbingeffect via a second output device to notify the external observer of anintent to convey a message, prior to conveying the human understandableoutput; wherein the output device is a first output device.
 25. Themethod of claim 24, wherein the first output device is the second outputdevice.
 26. The method of claim 24, wherein producing the attentiongrabbing effect includes displaying a notification on a ground surfacein proximity to the autonomous vehicle.
 27. The method of claim 24,further comprising detecting a location of the external observer basedon the collected data; and wherein producing the attention grabbingeffect includes projecting light towards a direction corresponding tothe detected location of the external observer.
 28. The method of claim19, wherein determining the intended course of action includesgenerating a request to deliver to the external observer; and whereinconveying the human understandable output includes presenting therequest to the external observer.
 29. The method of claim 19, whereinconveying the human understandable output includes indicating that theautonomous vehicle is in a convoy.
 30. The method of claim 19, whereindetermining the intended course of action includes determining that theautonomous vehicle is to enter into a convoy; and wherein conveying thehuman understandable output includes indicating that the autonomousvehicle is waiting for or ready to enter into the convoy.
 31. The methodof claim 19, wherein the human understandable output is a movement stateindicator.
 32. The method of claim 31, wherein conveying the humanunderstandable output includes displaying the movement state indicatoron a ground surface in proximity to the autonomous vehicle.
 33. Themethod of claim 32, wherein the movement state indicator takes a form ofa line indicating that the autonomous vehicle will not cross the line.34. The method of claim 31, wherein the movement state indicatorincludes a stopped state, a normal speed state, and a reduced speedstate.
 35. The method of claim 19, wherein conveying the humanunderstandable output includes embedding machine readable information inthe human understandable output.
 36. The method of claim 35, wherein themachine readable information conveys additional information beyond thatconveyed in the human understandable output.
 37. The method of claim 35,wherein embedding the machine readable information includes pulse-codingthe human understandable output.